Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: erasure decoder #4448

Merged
merged 45 commits into from
Dec 4, 2023
Merged

feat: erasure decoder #4448

merged 45 commits into from
Dec 4, 2023

Conversation

nugaon
Copy link
Member

@nugaon nugaon commented Nov 7, 2023

Creating redundancy getter which retrieves children chunks of an intermediate chunk and tries to recover missing data if there is any.

Its initialization happens by passing the data and parity shard addresses of the intermediate chunk having some parities greater than 0. To extract these addresses ChunkAddresses function was created in pkg/file/utils.go. along with ChunkPayloadSize function that identifies the effective size of an intermediate chunk. The redundancy level is encoded in the span of the intermediate chunk which is used to identify shard and parity reference counts in a chunk.
The getter also requires a storage.Putter interface for saving recovered chunks to the localStorage.

Let n indicate the data shard number and p for the parity number.
If a chunk is missing (or the faster requests strategy is enabled) then the recovery process will start n + k requests for data and parity shards and cancel all requests if n requests returned successfully. Data shards not returned in this process will be recovered and saved.

The redundancy getter is used by the file joiner that is responsible to read a file.

what is left:

  • apply different strategies to the getter

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

@nugaon nugaon changed the base branch from master to feat/erasure-encoder November 7, 2023 21:24
@nugaon nugaon force-pushed the feat/erasure-encoder branch 2 times, most recently from cb39e6b to 5bb42e0 Compare November 8, 2023 11:47
@nugaon nugaon force-pushed the feat/erasure-decoder branch from d42c28e to 51948f1 Compare November 13, 2023 09:51
@nugaon nugaon force-pushed the feat/erasure-decoder branch from 8d7264a to 0d54a70 Compare December 1, 2023 09:13
@nugaon nugaon marked this pull request as ready for review December 3, 2023 16:30
Base automatically changed from feat/erasure-encoder to feat/redundancy December 4, 2023 14:40
@nugaon nugaon mentioned this pull request Dec 4, 2023
7 tasks
@nugaon nugaon force-pushed the feat/erasure-decoder branch from e7d9907 to 9cd6f19 Compare December 4, 2023 15:00
@nugaon nugaon merged commit cacdbd6 into feat/redundancy Dec 4, 2023
13 checks passed
@nugaon nugaon deleted the feat/erasure-decoder branch December 4, 2023 15:23
nugaon added a commit that referenced this pull request Dec 4, 2023
nugaon added a commit that referenced this pull request Dec 18, 2023
nugaon added a commit that referenced this pull request Dec 20, 2023
nugaon added a commit that referenced this pull request Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant